Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control.

Objective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Vision sensors are a major source of information about the environment state and can play a vital role in inferring feasible and intended gestures. However, visual evidence is also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. Multimodal evidence fusion using physiological and vision sensor measurements is a natural approach due to the complementary strengths of these modalities. Methods: In this paper, we present a Bayesian evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. We analyze individual and fused performance as a function of time as the hand approaches the object to grasp it. For this purpose, we have also developed novel data processing and augmentation techniques to train neural network components. Results: Our results indicate that, on average, fusion improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%. Conclusion: Our experimental data analyses demonstrate that EMG and visual evidence show complementary strengths, and as a consequence, fusion of multimodal evidence can outperform each individual evidence modality at any given time.

Investigating and acquiring motor expertise using virtual reality.

After just months of simulated training, on January 19, 2019 a 23-year-old E-sports pro-gamer, Enzo Bonito, took to the racetrack and beat Lucas di Grassi, a Formula E and ex-Formula 1 driver with decades of real-world racing experience. This event raised the possibility that practicing in virtual reality can be surprisingly effective for acquiring motor expertise in real-world tasks. Here, we evaluate the potential of virtual reality to serve as a space for training to expert levels in highly complex real-world tasks in time windows much shorter than those required in the real world and at much lower financial cost without the hazards of the real world. We also discuss how VR can also serve as an experimental platform for exploring the science of expertise more generally.

The Impact of Fatigue on the Sense of Local and Global Rhythmic Movement.

Efficient movement control and the mechanisms responsible for the sense of rhythm are still not fully understood. The purpose of this paper was to estimate the influence of fatigue on the sense of rhythm defined as specific order of movements and their rhythmic perception. It was examined in a holistic way, by analyzing both global and local aspects of the movement. Twenty adult participants (20.2 ± 0.4 years, ten females) took part in the experiment. The fatigue protocol was applied in four blocks, which consisted of 30-s consecutive jumping with 80% of maximal effort. Immediately after each fatigue block, the rhythm performance was evaluated in global and local tests. The global test was based on 45 continuous jumps and was divided into an assisted and an unassisted phase using the Optojump Next System. The local test was performed by bilateral tapping of lower limbs by means of the Vienna Test System. The hypothesis about the significant effect of fatigue on the sense of rhythm was falsified. In particular, we observed the lack of differences between global and local aspects of the movement. Moreover, female participants showed a better sense of rhythm than males. Regardless of the fatigue protocol, participants made larger errors with a lower movement frequency in local rhythmic tasks. The coefficient of variation showed that sex differences were only significant in the unassisted phase of the global rhythmic task. We suggest that movement variability metrics may provide additional information about the sense of rhythm, which should be explored more in future studies, not only dependent on fatigue.

A kinematic and EMG dataset of online adjustment of reach-to-grasp movements to visual perturbations.

Control of reach-to-grasp movements for deft and robust interactions with objects requires rapid sensorimotor updating that enables online adjustments to changing external goals (e.g., perturbations or instability of objects we interact with). Rarely do we appreciate the remarkable coordination in reach-to-grasp, until control becomes impaired by neurological injuries such as stroke, neurodegenerative diseases, or even aging. Modeling online control of human reach-to-grasp movements is a challenging problem but fundamental to several domains, including behavioral and computational neuroscience, neurorehabilitation, neural prostheses, and robotics. Currently, there are no publicly available datasets that include online adjustment of reach-to-grasp movements to object perturbations. This work aims to advance modeling efforts of reach-to-grasp movements by making publicly available a large kinematic and EMG dataset of online adjustment of reach-to-grasp movements to instantaneous perturbations of object size and distance performed in immersive haptic-free virtual environment (hf-VE). The presented dataset is composed of a large number of perturbation types (10 for both object size and distance) applied at three different latencies after the start of the movement.

Classifications of Dynamic EMG in Hand Gesture and Unsupervised Grasp Motion Segmentation.

The electromyography (EMG) signals have been widely utilized in human-robot interaction for extracting user hand/arm motion instructions. A major challenge of the online interaction with robots is the reliable EMG recognition from real-time data. However, previous studies mainly focused on using steady-state EMG signals with a small number of grasp patterns to implement classification algorithms, which is insufficient to generate robust control regarding the dynamic muscular activity variation in practice. Introducing more EMG variability during training and validation could implement a better dynamic-motion detection, but only limited research focused on such grasp-movement identification, and all of those assessments on the non-static EMG classification require supervised ground-truth label of the movement status. In this study, we propose a framework for classifying EMG signals generated from continuous grasp movements with variations on dynamic arm/hand postures, using an unsupervised motion status segmentation method. We collected data from large gesture vocabularies with multiple dynamic motion phases to encode the transitions from one intent to another based on common sequences of the grasp movements. Two classifiers were constructed for identifying the motion-phase label and grasptype label, where the dynamic motion phases were segmented and labeled in an unsupervised manner. The proposed framework was evaluated in real-time with the accuracy variation over time presented, which was shown to be efficient due to the high degree of freedom of the EMG data.

Control of aperture closure during reach-to-grasp movements in immersive haptic-free virtual reality.

Virtual reality (VR) has garnered much interest as a training environment for motor skill acquisition, including for neurological rehabilitation of upper extremities. While the focus has been on gross upper limb motion, VR applications that involve reaching for, and interacting with, virtual objects are growing. The absence of true haptics in VR when it comes to hand-object interactions raises a fundamentally important question: can haptic-free immersive virtual environments (hf-VEs) support naturalistic coordination of reach-to-grasp movements? This issue has been grossly understudied, and yet is of significant importance in the development and application of VR across a number of sectors. In a previous study (Furmanek et al., J Neuroeng Rehabil 16:78, 2019), we reported that reach-to-grasp movements are similarly coordinated in both the physical environment (PE) and hf-VE. The most noteworthy difference was that the closure phase-which begins at maximum aperture and lasts through the end of the movement-was longer in hf-VE than in PE, suggesting that different control laws might govern the initiation of closure between the two environments. To do so, we reanalyzed data from Furmanek et al. (J Neuroeng Rehabil 16:78, 2019), in which the participants reached to grasp three differently sized physical objects, and matching 3D virtual object renderings, placed at three different locations. Our analysis revealed two key findings pertaining to the initiation of closure in PE and hf-VE. First, the respective control laws governing the initiation of aperture closure in PE and hf-VE both included state estimates of transport velocity and acceleration, supporting a general unified control policy for implementing reach-to-grasp across physical and virtual environments. Second, the aperture was less informative to the control law in hf-VE. We suggest that the latter was likely because transport velocity at closure onset and aperture at closure onset were less independent in hf-VE than in PE, ultimately resulting in an aperture at closure onset having a weaker influence on the initiation of closure. In this way, the excess time and muscular effort needed to actively bring the fingers to a stop at the interface of a virtual object was factored into the control law governing the initiation of closure in hf-VE. Critically, this control law remained applicable, albeit with different weights in hf-VE, despite the absence of terminal haptic feedback and potential perceptual differences.

Synergistic Activation Patterns of Hand Muscles in Left-and Right-Hand Dominant Individuals.

Handedness has been associated with behavioral asymmetries between limbs that suggest specialized function of dominant and non-dominant hand. Whether patterns of muscle co-activation, representing muscle synergies, also differ between the limbs remains an open question. Previous investigations of proximal upper limb muscle synergies have reported little evidence of limb asymmetry; however, whether the same is true of the distal upper limb and hand remains unknown. This study compared forearm and hand muscle synergies between the dominant and non-dominant limb of left-handed and right-handed participants. Participants formed their hands into the postures of the American Sign Language (ASL) alphabet, while EMG was recorded from hand and forearm muscles. Muscle synergies were extracted for each limb individually by applying non-negative-matrix-factorization (NMF). Extracted synergies were compared between limbs for each individual, and between individuals to assess within and across participant differences. Results indicate no difference between the limbs for individuals, but differences in limb synergies at the population level. Left limb synergies were found to be more similar than right limb synergies across left- and right-handed individuals. Synergies of the left hand of left dominant individuals were found to have greater population level similarity than the other limbs tested. Results are interpreted with respect to known differences in the neuroanatomy and neurophysiology of proximal and distal upper limb motor control. Implications for skill training in sports requiring dexterous control of the hand are discussed.

Number of Trials Necessary to Apply Analysis Within the Framework of the Uncontrolled Manifold Hypothesis at Different Levels of Hierarchical Synergy Control.

The uncontrolled manifold hypothesis is a method used to quantify motor synergies, defined as a specific central nervous system organization that maintains the task-specific stability of motor actions. The UCM allows for inter-trial variance analysis between consecutive trials. However, despite the large body of literature within this framework, there is no report on the number of movement repetitions required for reliable results. Based on the hypothetical hierarchical control of motor synergies, this study aims to determine the minimum number of trials necessary to achieve a good to excellent level of reliability. Thirteen young, healthy participants performed fifteen bilateral isometric contractions of elbow flexion when visual feedback was provided. The force and electromyography data were recorded to investigate synergies at different levels of hierarchical control. The intraclass correlation coefficient was used to determine the reliability of the variance indices. Based on the obtained results, at least twelve trials are required to analyze the inter-trial variance in both force and muscle synergies within the UCM framework.

Reliability and Validity of the Trunk Control Measurement Scale Among Children and Adolescents With Cerebral Palsy in Tanzania.

The aim of this study was to investigate the reliability and validity of the Trunk Control Measurement Scale (TCMS) among children with cerebral palsy (CP) who were living in Tanzania. Two physiotherapy trainees, each blinded to the other's test findings, independently administered the TCMS to 38 children with CP (16 female; M age = 7.2, SD = 4.8 years) twice over sessions separated by a 30-day interval. We assessed internal consistency and intra/inter-rater reliability using Cronbach's alpha and the Intraclass Correlation Coefficient (ICC), respectively. We examined measurement error through Standard Error of Measurement (SEM) and Minimal Detectable Change (MDC). We assessed construct validity with Spearman's correlation coefficient between the TCMS and both the Gross Motor Function Classification System (GMFCS) and the Manual Ability Classification System (MACS). We found a frank floor effect. Internal consistency (α = 0.945) and reliability indices were excellent for the total scale (ICC for intra-rater reliability = 0.985, inter- reliability = 0.997) and for each sub-scale score. We found low values of SEM (1.7 points) and MDC (4.8 points). Construct validity was supported by moderate and strong correlations between the TCMS and the GMFCS and MACS in this sample. We conclude that the psychometric properties of the TMCS support its clinical and research use for youth with CP in low-income settings.

Postural constraints recruit shorter-timescale processes into the non-Gaussian cascade processes.

Healthy human postural sway exhibits strong intermittency, reflecting a richly interactive foundation of postural control. From a linear perspective, intermittent fluctuations might be interpreted as engagement and disengagement of complementary control processes at distinct timescales or from a nonlinear perspective, as cascade-like interactions across many timescales at once. The diverse control processes entailed by cascade-like multiplicative dynamics suggest specific non-Gaussian distributional properties at different timescales. Multiscale probability density function (PDF) analysis showed that when standing quietly while balancing a sand-filled tube with the two arms elicited non-Gaussianity profiles showing a negative-quadratic crossover between short and long timescales. A more stringent task of balancing a water-filled tube elicited simpler monotonic decreases in non-Gaussianity, that is, a positive-quadratic cancellation of the negative-quadratic crossover. Multiple known indices of postural sway governed the appearance or disappearance of the crossover. Finally, both tasks elicited lognormal distributions over progressively larger timescales. These results provide the first evidence that more stringent postural constraints recruit shorter-timescale processes into the non-Gaussian cascade processes, that indices of postural sway moderate this recruitment, and that more stringent postural constraints show stronger statistical hallmarks of cascade structure.

Movement Quality: A Novel Biomarker Based on Principles of Neuroscience.

A major problem in neurorehabilitation is the lack of objective outcomes to measure movement quality. Movement quality features, such as coordination and stability, are essential for everyday motor actions. These features allow reacting to continuously changing environment or to resist external perturbations. Neurological disorders affect movement quality, leading to functionally impaired movements. Recent findings suggest that the central nervous system organizes motor elements (eg, muscles, joints, fingers) into task-specific ensembles to stabilize motor tasks performance. A method to quantify this feature has been previously developed based on the uncontrolled manifold (UCM) hypothesis. UCM quantifies movement quality in a spatial-temporal domain using intertrial analysis of covariation between motor elements. In this point-of-view article, we first describe major obstacles (eg, the need for group analysis) that interfere with UCM application in clinical settings. Then, we propose a process of quantifying movement quality for a single individual with a novel use of bootstrapping simulations and UCM analysis. Finally, we reanalyze previously published data from individuals with neurological disorders performing a wide range of motor tasks, that is, multi-digit pressing and postural balance tasks. Our method allows one to assess motor quality impairments in a single individual and to detect clinically important motor behavior changes. Our solution may be incorporated into a clinical setting to assess sensorimotor impairments, evaluate the effects of specific neurological treatments, or track movement quality recovery over time. We also recommended the proposed solution to be used jointly with a typical statistical analysis of UCM parameters in cohort studies.

Coordination of reach-to-grasp in physical and haptic-free virtual environments.

BACKGROUND: Virtual reality (VR) offers unprecedented opportunity as a scientific tool to study visuomotor interactions, training, and rehabilitation applications. However, it remains unclear if haptic-free hand-object interactions in a virtual environment (VE) may differ from those performed in the physical environment (PE). We therefore sought to establish if the coordination structure between the transport and grasp components remain similar whether a reach-to-grasp movement is performed in PE and VE. METHOD: Reach-to-grasp kinematics were examined in 13 healthy right-handed young adults. Subjects were instructed to reach-to-grasp-to-lift three differently sized rectangular objects located at three different distances from the starting position. Object size and location were matched between the two environments. Contact with the virtual objects was based on a custom collision detection algorithm. Differences between the environments were evaluated by comparing movement kinematics of the transport and grasp components. RESULTS: Correlation coefficients, and the slope of the regression lines, between the reach and grasp components were similar for the two environments. Likewise, the kinematic profiles of the transport velocity and grasp aperture were strongly correlated across the two environments. A rmANOVA further identified some similarities and differences in the movement kinematics between the two environments - most prominently that the closure phase of reach-to-grasp movement was prolonged when movements were performed in VE. CONCLUSIONS: Reach-to-grasp movement patterns performed in a VE showed both similarities and specific differences compared to those performed in PE. Additionally, we demonstrate a novel approach for parsing the reach-to-grasp movement into three phases- initiation, shaping, closure- based on established kinematic variables, and demonstrate that the differences in performance between the environments are attributed to the closure phase. We discuss this in the context of how collision detection parameters may modify hand-object interactions in VE. Our study shows that haptic-free VE may be a useful platform to study reach-to-grasp movements, with potential implications for haptic-free VR in neurorehabilitation.

The Effects of Cryotherapy on Knee Joint Position Sense and Force Production Sense in Healthy Individuals.

The proprioceptive information received from mechanoreceptors is potentially responsible for controlling the joint position and force differentiation. However, it is unknown whether cryotherapy influences this complex mechanism. Previously reported results are not universally conclusive and sometimes even contradictory. The main objective of this study was to investigate the impact of local cryotherapy on knee joint position sense (JPS) and force production sense (FPS). The study group consisted of 55 healthy participants (age: 21 ± 2 years, body height: 171.2 ± 9 cm, body mass: 63.3 ± 12 kg, BMI: 21.5 ± 2.6). Local cooling was achieved with the use of gel-packs cooled to -2 ± 2.5°C and applied simultaneously over the knee joint and the quadriceps femoris muscle for 20 minutes. JPS and FPS were evaluated using the Biodex System 4 Pro apparatus. Repeated measures analysis of variance (ANOVA) did not show any statistically significant changes of the JPS and FPS under application of cryotherapy for all analyzed variables: the JPS's absolute error (p = 0.976), its relative error (p = 0.295), and its variable error (p = 0.489); the FPS's absolute error (p = 0.688), its relative error (p = 0.193), and its variable error (p = 0.123). The results indicate that local cooling does not affect proprioceptive acuity of the healthy knee joint. They also suggest that local limited cooling before physical activity at low velocity did not present health or injury risk in this particular study group.

Synergies and Motor Equivalence in Voluntary Sway Tasks: The Effects of Visual and Mechanical Constraints.

The authors used two analyses developed within the framework of the uncontrolled manifold hypothesis to quantify multimuscle synergies during voluntary body sway: analysis of intertrial variance and analysis of motor equivalence with respect to the center of pressure (COP) trajectory. Participants performed voluntary sway tasks in the anteroposterior direction at 0.33 and 0.66 Hz. Muscle groups were identified in the space of muscle activations and used as elemental variables in the synergy analyses. Changing mechanical and vision feedback-based constraints led to significant changes in indices of sway performance such as COP deviations in the uninstructed, mediolateral direction and indices of spontaneous postural sway. In contrast, there were no significant effects on synergy indices. These findings show that the neural control of performance and of its stability may involve different control variables and neurophysiological structures. There were strong correlations between the indices of motor equivalence and those computed using the intercycle variance analysis. This result is potentially important for studies of patients with movement disorders who may be unable to perform multiple trials (cycles) at any given task, making analysis of motor equivalence of single trials a viable alternative to explore changes in stability of actions.

Unintentional drifts during quiet stance and voluntary body sway.

We explored unintentional drifts in voluntary whole-body sway tasks following the removal of visual feedback. The main hypothesis was that the unintentional drifts were produced by drifts of referent coordinates for salient performance variables. Young healthy subjects stood quietly on a force platform and also performed voluntary body sway at 0.5 Hz both in the anterio-posterior and medio-lateral directions. Visual feedback on the center of pressure (COP) coordinate was provided and then turned off. During quiet stance trials, the subjects matched the initial COP coordinate with a target shifted by 3 cm anterior, posterior, left, or right from the coordinate during natural standing and activated the right tibialis anterior to 30% of its maximal voluntary contraction. During cyclical voluntary sway task, the nominal sway amplitude was always 4 cm while the midpoint was at one of the four mentioned locations. Removing visual feedback caused COP drifts during quiet stance trials that were consistent across trials performed by a subject but could be in opposite directions across subjects; there was a consistent drop in the activation level of tibialis anterior. During voluntary body sway, removing visual feedback caused a consistent increase in the voluntary sway amplitude and a drift of the midpoint that was consistent within but not across subjects. Motor equivalent and non-motor equivalent inter-cycle motion components were quantified within the space of muscle groups (muscle modes) under visual feedback and at the end of the period without visual feedback. Throughout the trial, there were large motor equivalent motion components, and they increased over the period without visual feedback. The results corroborate the idea that referent coordinate drifts at different levels of the control hierarchy can lead to unintentional drifts in performance. It suggests that directions of COP drifts are defined by two main factors, drift of the body referent coordinate toward the actual coordinate (that can lead to fall) and an opposite drift to ensure body motion to a safer location. Analysis of motor equivalence suggests that postural stability is not compromised during unintentional drifts in performance in contrast to earlier studies of multi-finger tasks. This may be due to the vital importance of postural stability for everyday actions.

Sensory Integration during Vibration of Postural Muscle Tendons When Pointing to a Memorized Target.

Vibrating ankle muscles in freely standing persons elicits a spatially oriented postural response. For instance, vibrating the Achilles tendons induces a backward displacement of the body while vibrating the tibialis anterior muscle tendons induces a forward displacement. These displacements have been called vibration induced falling (VIF) responses and they presumably are automatic. Because of the long delay between the onset of the vibration and the onset of the VIF (about 700 ms), and the widespread cortical activation following vibration, there is a possibility that the sensory signals available before the VIF can be used by the central nervous system to plan a hand pointing action. This study examined this suggestion. Ten healthy young participants stood on a force platform and initially were trained to point with and without vision to a target located in front of them. Then, they were exposed to conditions with vibration of the Achilles tendons or tibialis anterior muscle tendons and pointed at the target without vision. The vibration stopped between each trial. Trials with vision (without vibration) were given every five trials to maintain an accurate perception of the target's spatial location. Ankle vibrations did not have an effect on the position of the center of foot pressure (COP) before the onset of the pointing actions. Furthermore, reaction and movement times of the pointing actions were unaffected by the vibration. The hypotheses were that if proprioceptive information evoked by ankle vibrations alters the planning of a pointing action, the amplitude of the movement should scale according to the muscle tendons that are vibrated. For Achilles tendon vibration, participants undershot the target indicating the planning of the pointing action was influenced by the vibration-evoked proprioceptive information (forward displacement of the body). When the tibialis anterior were vibrated (backward displacement of the body), however, shorter movements were also observed. Longer movements would have increased the backward response of the sensed body movement. Thus, it is possible that pointing actions were adjusted on the basis of the expected consequences of the planned pointing action to avoid a response that could have compromised postural stability.